Convergence Analysis of the Approximate Newton Method for Markov Decision Processes

Recently two approximate Newton methods were proposed for the optimisation of Markov Decision Processes. While these methods were shown to have desirable properties, such as a guarantee that the preconditioner is negative-semidefinite when the policy is $\log$-concave with respect to the policy parameters, and were demonstrated to have strong empirical performance in challenging domains, such as the game of Tetris, no convergence analysis was provided. The purpose of this paper is to provide such an analysis. We start by providing a detailed analysis of the Hessian of a Markov Decision Process, which is formed of a negative-semidefinite component, a positive-semidefinite component and a remainder term. The first part of our analysis details how the negative-semidefinite and positive-semidefinite components relate to each other, and how these two terms contribute to the Hessian. The next part of our analysis shows that under certain conditions, relating to the richness of the policy class, the remainder term in the Hessian vanishes in the vicinity of a local optimum. Finally, we bound the behaviour of this remainder term in terms of the mixing time of the Markov chain induced by the policy parameters, where this part of the analysis is applicable over the entire parameter space. Given this analysis of the Hessian we then provide our local convergence analysis of the approximate Newton framework.Comment: This work has been removed because a more recent piece (A Gauss-Newton method for Markov Decision Processes, T. Furmston & G. Lever) of work has subsumed i

A Significance Test for Inferring Affiliation Networks from Spatio-Temporal Data.

Scientists have long been interested in studying social structures within groups of gregarious animals. However, obtaining evidence about interactions between members of a group is difficult. Recent technologies, such as Global Positioning System technology, have made it possible to obtain a vast wealth of animal movement data, but inferring the underlying (latent) social structure of the group from such data remains an important open problem. While intuitively appealing measures of social interaction exist in the literature, they typically lack formal statistical grounding. In this article, we provide a statistical approach to the problem of inferring the social structure of a group from the movement patterns of its members. By constructing an appropriate null model, we are able to construct a significance test to detect meaningful affiliations between members of the group. We demonstrate our method on large-scale real-world data sets of positional data of flocks of Merino sheep, Ovis aries

Pravastatin for early-onset pre-eclampsia:a randomised, blinded, placebo-controlled trial

Objective: Women with pre-eclampsia have elevated circulating levels of soluble fms-like tyrosine kinase-1 (sFlt-1). Statins can reduce sFlt-1 from cultured cells and improve pregnancy outcome in animals with a pre-eclampsia-like syndrome. We investigated the effect of pravastatin on plasma sFlt-1 levels during pre-eclampsia. Design: Blinded (clinician and participant), proof of principle, placebo-controlled trial. Setting: Fifteen UK maternity units. Population: We used a minimisation algorithm to assign 62 women with early-onset pre-eclampsia (24 +0–31 +6 weeks of gestation) to receive pravastatin 40 mg daily (n = 30) or matched placebo (n = 32), from randomisation to childbirth. Primary outcome: Difference in mean plasma sFlt-1 levels over the first 3 days following randomisation. Results: The difference in the mean maternal plasma sFlt-1 levels over the first 3 days after randomisation between the pravastatin (n = 27) and placebo (n = 29) groups was 292 pg/ml (95% CI −1175 to 592; P = 0.5), and over days 1–14 was 48 pg/ml (95% CI −1009 to 913; P = 0.9). Women who received pravastatin had a similar length of pregnancy following randomisation compared with those who received placebo (hazard ratio 0.84; 95% CI 0.50–1.40; P = 0.6). The median time from randomisation to childbirth was 9 days [interquartile range (IQR) 5–14 days] for the pravastatin group and 7 days (IQR 4–11 days) for the placebo group. There were three perinatal deaths in the placebo-treated group and no deaths or serious adverse events attributable to pravastatin. Conclusions: We found no evidence that pravastatin lowered maternal plasma sFlt-1 levels once early-onset pre-eclampsia had developed. Pravastatin appears to have no adverse perinatal effects. Tweetable abstract: Pravastatin does not improve maternal plasma sFlt-1 or placental growth factor levels following a diagnosis of early preterm pre-eclampsia #clinicaltrial finds

Research data supporting "A Significance Test for Inferring Affiliation Networks from Spatio-Temporal Data"

Movement patterns of several flocks of Merrino sheep (Ovis aries). The size of the flocks vary in size, from 11 animals up to around 100 animals. The movement patterns were obtained through GPS technology. The data sets are each around 24 hours in length, with the positional location of the animals given at a rate of 1Hz. The data and the code are provided as part of the publication "A Significance Test for Inferring Affiliation Networks from Spatio-Temporal Data" in the journal, PLOS ONE.This work was supported by CHD